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There is ample discussion in the literature concerning the ideal placement of 
demographic questions in survey research. Many argue that demographic questions 
should not be placed at the start of the survey because doing so can deter respondents 
and decrease response rates (Stoutenbourgh 2008). In addition, asking individuals 
about their demographic characteristics at the end of a survey may be preferable 
because if asked at the start, individuals’ subsequent performance or responses could 
be impacted (Gehlbach and Artino 2018). This has been documented in the area of 
cognitive testing in lab settings. Research shows that activating a stereotype can elicit 
behavior that is consistent with the activated stereotype (Steele and Aronson 1995). For 
example, priming women prior to a math test may activate the stereotype that women 
are relatively bad at math, thereby handicapping their performance (Spencer, Steele, 
and Quinn 1999). Notably, however, this effect seems to be limited to lab studies and 

is not an issue in real-world, high-stakes testing settings (Shewach, Sackett, and Quint 
2019). 


Little research has been done in areas related to social and emotional learning to 
confirm the hypothesis that making participants’ demographic characteristics salient 
activates stereotype threats, which then impact item responses (Gehlbach and Artino 
2018). In two studies outside the area of cognitive testing, one examining employees’ 
ratings of workplace characteristics (Teclaw, Price, and Osatuke 2012) and one 
examining individuals’ reporting of socially desirable behavior (Frick, Bachtiger, and 
Reips 1999), no mean-level differences were observed between responses given 
subsequent to versus prior to demographic questions. This suggests that placing 
demographic questions at the beginning of a survey does not impact responses. 


In the current study, we extended this research to specifically examine whether placing 
demographic questions at the beginning of a survey impacts student responses to a 
social and emotional skills assessment. 
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Key Findings 


* Overall, we conclude that placing demographic questions at the start of a social 
and emotional skill assessment does not affect the nature of participant responses. 


¢ Regardless of one’s race, priming individuals with race had no effect on their social 
and emotional skill scores, and there was no indication that any stereotypes about 
their race were activated. 


¢« Regardless of one’s gender, priming individuals with gender had no effect on their 
social and emotional skill scores, and there was no indication that any stereotypes 
about their gender were activated. 


¢ Priming individuals with their academic performance had no effect on their 
social and emotional skill scores, and there was no indication that any related 
stereotypes were activated. 


Method 


Participants were students who took the ACT® test on the national testing date 

in December 2019 and responded to an invitation to participate in research after 
completing the ACT. Participants were removed from the data file if they were missing 
two or more social and emotional skill items or if they had zero variance on these 
items, suggesting careless responding (n = 278)." The final sample reported on 

here includes 1,789 students. Demographic information was obtained from the ACT 
registration file. Race/ethnicity (hereafter referred to simply as race) was unknown for 
some students, and the sample sizes of some racial groups were too small to make 
meaningful comparisons, so not all groups were included in analyses concerning 
race. Included in the analyses below are Black/African American (n = 181), White (n 
= 1,097), Hispanic/Latino (n = 195), and Asian (n = 110) students, as well as students 
who identified as two or more races (n = 89). Male (n = 482) and female (n = 1,247) 
students were also compared. The 60 students who identified as something other 
than male or female or whose gender was unknown were not included in analyses 
pertaining to gender, given that they would constitute a small and heterogeneous 
group, rendering our dichotomous gender comparison noncomprehensive. 


Students completed the survey in Qualtrics and were randomly assigned to one of five 
conditions: 


1. report race at the start of the survey, 
report gender at the start of the survey, 
report race and gender at the start of the survey, 


report GPA at the start of the survey, or 


ae SS 


report no demographic information. 


For analyses reported below, when race priming was the independent variable, groups 
1 and 3 were combined and contrasted with groups 2, 4, and 5 combined. When 
gender priming was the independent variable, groups 2 and 3 were combined and 
contrasted with groups 1, 4, and 5 combined.? 
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The students completed Likert measures of six social and emotional skills including 
Sustaining Effort, Getting Along with Others, Maintaining Composure, Keeping an 
Open Mind, Social Connection (ACT 2020), and Growth Mindset (Dweck 1999), as 

well as items designed to gauge the extent to which stereotype threats were activated. 
Activation of stereotype threat was evaluated with two Likert measures assessing 
feelings of during-survey anxiety and cognitive load (Hart and Staveland 1988) and 
single questions about the extent to which the respondent was thinking about his or her 
race (or gender) while answering questions in the survey and how the respondent felt 
about his or her race (or gender) while answering questions in the survey. 


Analyses and Results 


To determine whether priming race or gender had an effect on social and emotional skill 
responses and whether this varied according to racial or gender group, we carried out 
two-way MANOVAs. For completeness, we report the main effects for race and gender 
here, but, given that that was not central to our research questions, we do not elaborate 
on those findings. We carried out a second two-way MANOVA with the stereotype threat 
activation items as the dependent variables. In addition to examining effects of priming 
race and gender, we carried out one-way MANOVAs to examine the effect of priming 
academic performance, specifically GPA. All findings are reported in Table 1. 


Table 1. Effects of Priming Demographic Characteristics on Social and Emotional Skill Scores and 


Stereotype Threat Activation 


Wilks’ A ia df 
Race / Social and Emotional Skills 
Race .96 2.79 24, 5328 
Race Primed Condition 1.00 .80 6, 1527 
Interaction .98 1.03 24, 5328 
Race / Stereotype Threat Activation 
Race 91 8.63 16, 4195 
Race Primed Condition 1.00 16 4, 1373 
Interaction .98 1253 16, 4195 
Gender / Social and Emotional Skills 
Gender 93 19.23 6, 1584 
Gender Primed Condition 1.00 .86 6, 1584 
Interaction 1.00 .90 6, 1584 
Gender / Stereotype Threat Activation 
Gender .99 4.17 4, 1423 
Gender Primed Condition 1.00 18 4, 1423 
Interaction 1.00 22 4, 1423 
GPA / Social and Emotional Skills 
GPA Primed Condition 1.00 ios 6, 1644 
GPA / Stereotype Threat Activation 
GPA Primed Condition 1.00 30 2, 1483 


Note. Some df were non-integers and were rounded to the nearest whole number. 
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Discussion and Conclusions 


Anumber of arguments have been put forward for and against placing demographic 
items at the start of surveys, though no studies to date have specifically examined 
whether responding to demographic items activates stereotype threats, which in turn 
impact responses to social and emotional learning assessments. Determining the 
effects of demographic item placement is important to ensure that participation rates 
and responses are not compromised. Gathering demographic information is vital for 
describing the study sample in order to evaluate generalizability. Moreover, often, the 
demographic variables themselves are key study variables—for instance, in cases of 
examining subgroup differences. Therefore, it would be advantageous to collect these 
data early in the study provided it does not jeopardize data quality. 


In the current study, there were no significant main effects for any of the experimental 
conditions. Likewise, there were no significant interactions. That is, priming individuals 
with race or gender had no effect on their social and emotional skill scores, and there 
was no indication that any stereotypes about their race or gender were activated. This 
was true regardless of a respondent's race or gender. Priming individuals with their 
academic performance had no effect on their social and emotional skill scores, and there 
was no indication that any related stereotypes were activated. Thus, we conclude that 
placing demographic questions at the start of a social and emotional skill assessment 
does not affect the nature of participant responses. 
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Notes 


1. We assessed whether placement of demographic items had an impact on participants’ 
willingness to complete the survey. There was no association between experimental 
condition and dropout, Xa) = 32, p = .99. 


2. Results were unchanged when examining each individual experimental group (i.¢., groups 
1, 2, 3, and 4) vs. the control group (i.e., group 5), indicating no confounding of the prime. 
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